As more and more data is hosted on the Internet, new techniques have been developed for sorting, organizing, and accessing this information. One common tool is the Internet search engine. Search engines allow a user to input a text query and receive a list of results associated with that query. The user may then access the result they deem most relevant to their query to receive additional information.
However, not all searches can be easily represented by a text query. For example, a user may wish to identify a person in a photograph, an actor in a video, or a location on a map. Such scenarios require the user to identify a text query that approximates the contents of their image, often leading to inaccurate results. Other search engines may offer an image based search option, where a user may upload a target image to initiate a search based on features of the image. However, such search engines typically operate by identifying a text query associated with the target image, and then performing a search based on the text query. For example, a user might notice a large cat in their backyard, and take a photograph of the animal. Wishing to identify the type of animal, the user may perform an image search by uploading the photograph to an image search engine. The photograph may be associated with the text “jaguar,” describing the large cat of the same name. However, the query might return results for the British car manufacturer JAGUAR, the sports team the JACKSONVILLE JAGUARS, and the animal panthera onca, commonly known as the jaguar. Although the context of the target image might indicate that the user wishes for more information on the cat, performing an image search in this manner does not properly account for the semantics or context of the image.